Recurrence of optimum for training weight and activation quantized networks

Deep neural networks (DNNs) are quantized for efficient inference on resource-constrained platforms. However, training deep learning models with low-precision weights and activations involves a demanding optimization task, which calls minimizing stage-wise loss function subject to discrete set-constraint. While numerous methods have been proposed, existing studies full quantization of DNNs mostly empirical. From theoretical point view, we study practical techniques overcoming the combinatorial nature network quantization. Specifically, investigate simple yet powerful projected gradient-like algorithm quantizing two-layer convolutional networks, by repeatedly moving one step at float in negative direction heuristic fake gradient (so-called coarse gradient) evaluated weights. For first time, prove that under mild conditions, sequence recurrently visit global optimum minimization problem fully network. We also show numerical evidence recurrence phenomenon weight evolution networks.